Setting general connection properties

This section describes how to configure general connection properties. For an explanation of how to configure advanced connection properties, see Setting advanced connection properties.

To define the general connection properties:

1. Click the Manage Endpoint Connections toolbar button.

   The Manage Endpoints Connections dialog box opens.

2. Click the New Endpoint Connection toolbar button.

   The Name, Description, Type and Role fields are displayed on the right.

3. In the Name field, specify a display name for the endpoint.
4. In the Description field, optionally type a description for the Amazon MSK endpoint.
5. Select Target as the endpoint Role.

6. Select Amazon MSKas the endpoint Type.

   The dialog box is divided into General and Advanced tabs.

7. In the Broker servers field, specify one or more broker servers using the following format (for high availability):

   server1[:port1][,server2[:port2]]

   Example:

   192.168.1.100:9092,192.168.1.101:9093

   Replicate will connect to the first available host. If a host is specified without a port then port 9092 will be used as the default.

   Information note

   When using SSL authentication, you must specify the broker FQDN (i.e. not the IP address).

   Information note

   All of the broker servers in your cluster need to be accessible to Replicate. However, you do not need to specify all of the servers in the Broker servers field. This is because Replicate only need to connect to one of the servers in order to retrieve the connection details for the other servers in the cluster. It is therefore best practice to specify the servers that are most likely to be available when the task is run. The servers to which Replicate produces messages is determined by the topic and partitioning topic and partitioning settings described below.

8. Continue as described below.

 

Security

In the Security section, the following options can be set:

• Use SSL (supports TLS 1.2): Select this option to encrypt the communication between the Replicate machine and the broker server(s). If the brokers are configured to require SSL, then you must select this option.

• Authentication: Select one of the following:

  • None - No authentication.

  • Certificate - If you select this option, you also need to provide the following information:

    Note  The public and private key files must be in PEM format.

    • Public key file - The full path to the public key file on the Replicate Server machine.
    • Private key file - The full path to the private key file on the Replicate Server machine.
    • Private key password - The password for the private key file.

  • Username and Password (SASL/SCRAM-SHA-512) - You can select this option to authenticate yourself using a user name and password (SASL/SCRAM-SHA-512).

    Note that selecting this option also requires each broker's server.properties file to be configured with the corresponding SASL/SCRAM mechanism.

Message Properties

In the Message Properties section, set the following properties:

1. Choose JSON or Avro as the message format.

2. From the Compression drop-down list, optionally select one of the available compression methods (Snappy or gzip). The default is None.

Data Message Publishing

In the Data Message Publishing section, set the following properties:

1. In the Publish the data to field, choose one of the following:

   • Specific topic - to publish the data to a single topic. Either type a topic name or use the browse button to select the desired topic.

   • Specific topic for each table - to publish the data to multiple topics corresponding to the source table names.

     The target topic name consists of the source schema name and the source table name, separated by a period (e.g. "dbo.Employees"). The format of the target topic name is important as you will need to prepare these topics in advance.

   Information note

   If the topics do not exist, configure the brokers with auto.create.topics.enable=true to enable Replicate to create the topics during runtime. Otherwise, the task will fail.

2. From the Partition strategy drop-down list, field, select either Random or By message key. If you select Random, each message will be written to a randomly selected partition. If you select By message key, messages will be written to partitions based on the selected By message key (described below).

3. From the Message key drop-down list, field, select one of the following:

   • None - To create messages without a message key.

   • Schema and table name - For each message, the message key will contain a combination of schema and table name (e.g. "dbo+Employees").

     When By message key is selected as the Partition strategy, messages consisting of the same schema and table name will be written to the same partition.

   • Primary key columns - For each message, the message key will contain the value of the primary key column.

     When By message key is selected as the Partition strategy, messages consisting of the same primary key value will be written to the same partition.

Metadata Message Publishing

In the Metadata Message Publishing section, specify whether or where to publish the message metadata.

From the Publish drop-down list, select one of the following options:

• Do not publish metadata messages

  When this option is selected, only the data messages will be published.

• Publish data schemas to the Confluent Schema Registry

  If you select this option, you must also configure the Schema Registry Connection Properties described below.

Schema Registry Connection Properties

• Schema Registry servers: Specify one or more Schema Registry servers using the following format (for high availability):

  server1:port1[,server2[:port2]]

  Example:  

  192.168.1.100:8081,192.168.1.101:8081

  Replicate will connect to the first available host.

• Use SSL (supports TLS 1.0, 1.1 and 1.2): Select this option to encrypt the data between the Replicate machine and the Schema Registry server(s). If the servers are configured to require SSL, then you must select this option.

  • CA path: Specify one of the following:

    • The full path (i.e. including the file name) to a specific CA certificate in PEM format
    • The directory containing certificate files with hash names

• Authentication - Select one of the following Schema Registry authentication options:

  • None - No authentication.

  • Certificate - Select to authenticate using a certificate.

    If you select this option, you also need to provide the following information:

  • Public key file - The full path to the public key file on the Replicate Server machine.
  • Private key file - The full path to the private key file on the Replicate Server machine.
  • Private key password - The password for the private key file.

  • Username and Password - Select to authenticate with a username and password.

• Use proxy server - Select to publish to the Schema Registry via a proxy server.

  • Host name - The host name of the proxy server.
  • Port - The port via which to access the proxy server.
  • Scheme - Select which protocol to use to access the server (HTTP or HTTPS).
  • SSL CA Path: The location of the CA file on the Replicate Server machine when HTTPS is the selected Scheme.

Schema Registry Subject Properties

Subject Name Strategy

Select one of the available subject name strategies.

• Schema and Table Name - The default
• Topic Name
• Record Name
• Topic and Record Name - See also: Message Format.

For more information on Confluent's subject name strategies, see https://docs.confluent.io/platform/current/schema-registry/serdes-develop/index.html#subject-name-strategy

Subject Compatibility Mode

Select a compatibility mode from the Subject compatibility mode drop-down list. A description of the selected mode will appear below the drop-down list.